Method for gesture recognition, terminal, and storage medium

ABSTRACT

A method for gesture recognition, a terminal, and a storage medium are provided by the embodiments of the present application. The method may include: receiving, through the millimeter wave apparatus, a first millimeter wave, where the first millimeter wave is a reflected wave formed after a second millimeter wave transmitted by the millimeter wave apparatus is modulated via a gesture motion; processing the first millimeter wave based on two types of time arrays and Doppler estimation to obtain at least one set of signal characteristic values corresponding to the first millimeter wave; identifying the at least one set of signal characteristic values using a correspondence library of standard characteristic values and control instructions, and obtaining a first control instruction corresponding to the gesture motion; and controlling a first application to implement a corresponding function using the first control instruction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2018/103362, filed on Aug. 30, 2018, the disclosure of which ishereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present application relates to the field of electronic application,and in particular, to a method for gesture recognition, a terminal and astorage medium.

BACKGROUND

In recent years, with the rapid development of intelligent terminal, thefunctions of the terminal have become more and more abundant, and thecontrol of users on the terminal is not limited to the operation mannerssuch as clicking and sliding on the display interface of the terminal.In the future, gesture perception will become a development trend forthe control of the terminals by users, where the user does not need totouch the terminal, and only needs to change different gestures within acertain range of the terminal, so that the terminal recognizes differentgestures to implement corresponding different functions, which expandsthe manner of controlling of the terminal by users. The existing schemesof gesture recognition include sound wave gesture recognition andgesture recognition based on image analysis of visible light camera,which have the problem of a low accuracy of gesture recognition.

The solution of sound wave gesture recognition is taken as an example,in which the terminal reconstructs the gesture motion according to theultrasonic signal generated by a gesture motion of the user, however theaccuracy of the gesture recognition of the scheme of sound wave gesturerecognition in a noisy environment is greatly reduced. The scheme ofgesture recognition based on image analysis of visible light camera istaken as another example, in which the terminal reconstructs the gesturemotion according to a multi-angle gesture image captured by a camera,however, the accuracy of gesture recognition of this scheme isrelatively low in a dim light or zero light environment.

SUMMARY

The embodiments of the present application are intended to provide amethod for gesture recognition, a terminal, and a storage medium, whichcan improve the accuracy of gesture recognition.

The embodiment of the present application provides a method for gesturerecognition, which is applied to a terminal, where the terminal isprovided with a millimeter wave apparatus, and the method includes:

-   -   receiving, through the millimeter wave apparatus, a first        millimeter wave, where the first millimeter wave is a reflected        wave formed after a second millimeter wave transmitted by the        millimeter wave apparatus is modulated via a gesture motion;    -   processing the first millimeter wave based on two types of time        arrays and Doppler estimation to obtain at least one set of        signal characteristic values corresponding to the first        millimeter wave, where each set of signal characteristic values        of the at least one set of signal characteristic values        correspond to one frame of signal in the first millimeter wave;    -   identifying the at least one set of signal characteristic values        using a correspondence library of standard characteristic values        and control instructions, and obtaining a first control        instruction corresponding to the gesture motion; and    -   controlling a first application to implement a corresponding        function using the first control instruction.

An embodiment of the present application provides a terminal, where theterminal includes: a processor, a receiver, a memory, and acommunication bus, where the terminal is provided with a millimeter waveapparatus, and the receiver is configured to receive a first millimeterwave through the millimeter wave apparatus, where the first millimeterwave is a reflected wave formed after a second millimeter wavetransmitted by the millimeter wave apparatus is modulated via a gesturemotion; where the processor is configured to execute an operatingprogram stored in the memory to implement the following steps:

-   -   processing the first millimeter wave based on two types of time        arrays and Doppler estimation to obtain at least one set of        signal characteristic values corresponding to the first        millimeter wave, where each set of signal characteristic values        of the at least one set of signal characteristic values        correspond to one frame of signal in the first millimeter wave;        identifying the at least one set of signal characteristic values        using a correspondence library of standard characteristic values        and control instructions, and obtaining a first control        instruction corresponding to the gesture motion; and controlling        a first application to implement a corresponding function using        the first control instruction.

The embodiment of the present application provides a storage medium, onwhich a computer program is stored, and the storage medium is applied toa terminal, and when the computer program is executed by a processor,any one of the methods for gesture recognition as described above isimplemented.

The embodiments of the present application provide a method for gesturerecognition, a terminal, and a storage medium. The method includes:receiving, through the millimeter wave apparatus, a first millimeterwave, where the first millimeter wave is a reflected wave formed after asecond millimeter wave transmitted by the millimeter wave apparatus ismodulated via a gesture motion; processing the first millimeter wavebased on two types of time arrays and Doppler estimation to obtain atleast one set of signal characteristic values corresponding to the firstmillimeter wave, where each set of signal characteristic values of theat least one set of signal characteristic values correspond to one frameof signal in the first millimeter wave; identifying the at least one setof signal characteristic values using a correspondence library ofstandard characteristic values and control instructions, and obtaining afirst control instruction corresponding to the gesture motion; andcontrolling a first application to implement a corresponding functionusing the first control instruction. With the above solution, theterminal receives the first millimeter wave modulated by the gesturemotion through the millimeter wave apparatus, and processes the firstmillimeter wave based on the two types of time arrays according to thecharacteristic of small wavelength of the first millimeter wave, andobtains at least one set of signal characteristic values correspondingto the processed first millimeter wave using Doppler estimation, andfinally obtains the first control instruction corresponding to thegesture motion using the at least one set of signal characteristicvalues and the correspondence library of standard characteristic valuesand control instructions, thereby subtle gestures motion can beidentified, and the accuracy of gesture perception is improved.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a first flowchart of a method for gesture recognitionaccording to an embodiment of the present application;

FIG. 2 is a structural composition diagram of an exemplary terminalaccording to an embodiment of the present application;

FIG. 3 is a display diagram of an exemplary characteristic valuecorresponding to one frame of signal according to an embodiment of thepresent application;

FIG. 4 is an architecture diagram of an exemplary control of gestureaccording to an embodiment of the present application;

FIG. 5 is a second flowchart of a method for gesture recognitionaccording to an embodiment of the present application;

FIG. 6 is a schematic diagram of an exemplary Frequency ModulatedContinuous Wave according to an embodiment of the present application;

FIG. 7 is a schematic diagram of an exemplary Doppler shift according toan embodiment of the present application;

FIG. 8 is a schematic diagram of an exemplary convolutional neuralnetwork model according to an embodiment of the present application;

FIG. 9 is a flowchart of an exemplary gesture recognition based on aMatlab program according to an embodiment of the present application;

FIG. 10 is a schematic diagram of an exemplary gesture motion accordingto an embodiment of the present application; and

FIG. 11 is a schematic structural diagram of a terminal according to anembodiment of the present application.

DESCRIPTION OF EMBODIMENTS

In order to make the features and technical contents of the embodimentsof the present application be understood in more detail, theimplementations of the present application are elaborated in detail incombination with the accompanying drawings as follows, where theattached drawings are for illustrative purposes only and are not used tolimit the embodiments of the present application.

The millimeter wave refers to the frequency band of 30-300 GHz. Theabundance of available bandwidth of this frequency band results in alarge transmission rate when the data transmission is performed on thefrequency band of millimeter wave; the millimeter wave becomes thefrequency band of communication used by the 5^(th) generation (5G)wireless communication technology due to the characteristics of a largebandwidth and a high rate, and the wireless network rate can be greatlyincreased using millimeter waves to perform the data transmission. Forexample, IEEE 802.11ad operating in the frequency band of 60 GHzsupports the rate of data transmission up to 6.7 Gbps, while itsevolution standard IEEE 802.11ay will provide the rate of datatransmission of 20 Gbps. Therefore, millimeter wave radio is expected toenable wireless network to enter the multi-Gbps era. Therefore,millimeter wave radio module will be widely installed on mobile phones,wearables, smart hardware or a wider range of IoT (Internet of Things)devices, becoming a mainstream communication technology. In addition tothe high rate link, the characteristics of short wavelength, largebandwidth, and directional beams of the millimeter wave makehigh-resolution and highly robust human gesture perception possible.

Millimeter wave perception technology provides a smarter, moreconvenient, and more interesting application experience ofhuman-computer interaction. The basic principle is, transmitting, by amillimeter wave RF module, a millimeter wave, receiving, by a receivingmodule, a reflected wave of a gesture motion, and speculating the sizeof the distance, angle, velocity and energy of the gesture process viathe reflected wave, so as to perform the classification of the motions.The millimeter wave supports a variety of perception functions such asdistance measurement, gesture detection, approaching detection, peoplenumber detection, distance measurement, and existence detection, whichcan be applied in the following scenarios:

A ringtone scenario, such as the scenario of incoming calls and alarmclocks, where the user can lower the volume of the ringtone down to muteby a specific gesture (for example, close to the mobile phone).

A process of selfie, where a series of gestures can be used to “tell”the mobile phone the timing of photographing, the adjustment ofbrightness, the adjustment of focal distances, etc., thereby avoidingthe inconvenient operation of touching the screen of the mobile phone.

A leftward/rightward/upward/downward slide at the upper side of thescreen, to view a previous/next application, return to the desktop orenter a multitask.

An upward slide at the upper side of the screen, or a multi-finger pinchto enter the multitask or a specific mode.

Recognition of the fine gestures of the user near the frame of themobile phone to perform button operations such as hover slide, volumeadjustment and brightness adjustment of the video, music switch, andcamera filter switch.

A hover pat by a hand over the upward side of the screen to take ascreenshot.

A hover click performed to simulate the motion of screen clicking whenit is not convenient to click the screen (such as with hands in gloves),

Recognition of the movement locus of hand to add some video and photoeffects.

Long-distance photography, where gestures are used to switch camerafilters, adjust the focal distance of camera, pause and continue, deletecaptured content, and the like.

The following are scenarios in which the gesture recognition isperformed using the millimeter wave in a photographing scenarioaccording to an embodiment of the present application.

Embodiment 1

An embodiment of the present application provides a method for gesturerecognition, which is applied to a terminal provided with a millimeterwave apparatus. As shown in FIG. 1, the method may include:

S101. Receiving, through a millimeter wave apparatus, a first millimeterwave, where the first millimeter wave is a reflected wave formed after asecond millimeter wave transmitted by the millimeter wave apparatus ismodulated via a gesture motion.

The method for gesture recognition provided by the embodiment of thepresent application is applied to a scenario in which a gesture of auser is perceived to implement a contactless photography.

In the embodiment of the present application, the terminal may be anydevice having functions of communication and storage, such as a tabletcomputer, a mobile phone, an e-reader, a remote controller, a personalcomputer (PC), a notebook computer, an in-vehicle device, a networktelevision, and a wearable device. The terminal is specifically selectedaccording to the actual situation, and is not specifically limited inthis embodiment.

In the embodiment of the present application, the millimeter waveapparatus is disposed inside the screen of the terminal, and themillimeter wave apparatus includes a transmitting antenna and areceiving antenna.

It can be understood that since the millimeter wave radio can penetratenon-metal materials such as plastic, the millimeter wave apparatus ishidden and disposed inside the screen of the terminal, which does notchange the appearance of the terminal and is thus of great significancefor the shape design of the terminal.

In the embodiment of the present application, the terminal transmits awireless signal (second millimeter wave) through the transmittingantenna of the millimeter wave apparatus; a reflected signal (firstmillimeter wave) is formed after modulation via the hand motion in thetransmission range of the wireless signal; and then the transmittedsignal is captured by the receiving antenna of the millimeter waveapparatus.

In the embodiment of the present application, the terminal may transmitthe wireless signal through the transmitting antenna of the millimeterwave apparatus when the preset transmission time arrives, or theterminal may transmit the wireless signal through the transmittingantenna of the millimeter wave apparatus when a first application suchas a photographing application or a video shooting application islaunched, and the specific timing at which the terminal transmits thewireless signal through the transmitting antenna of the millimeter waveapparatus is selected according to the actual situation, which is notspecifically limited in the embodiment of the present application.

In the embodiment of the present application, the form of transmittingthe wireless signal by the millimeter wave apparatus is periodicallytransmitting Frequency Modulated Continuous Wave (FMCW), so that thefrequency changing rule of the first millimeter wave and that of thesecond millimeter wave are the same, which are both triangular waverule, however there is only a time difference in between, and theterminal can use this small time difference to calculate the targetdistance.

Exemplarily, as shown in FIG. 2, a Digital Signal Processing (DSP) isdisposed on the terminal, where the DSP is composed of four parts as adistance processing module, a Capon beam former, and an object detectionunit, and a Doppler estimation unit, where

Distance processing module: after the receiving antenna receives thereflected wave, the reflected wave is cached into an output cache areaof an Analog-to-Digital converter (ADC), and then the millimeter waveapparatus moves the reflected wave from the output cache area of the ADCto the local memory within the DSP. At this time, the distanceprocessing module performs a 16-bit fixed-point 1-D window and a FastFourier transform (FFT) of a 16-bit fixed-point 1-D, and transmits theresult to the Doppler estimation unit.

Capon beam former: configured to reconstruct a source signal from asensor array using formula (1)X(t)=A(θ)s(t)+n(t)   (1)

-   -   where s(t) is an input signal after mixing the baseband signals;    -   the static sundries is removed by removing the DC components of        each distance receiver in the distance processing module,        thereby eliminating reflections of static objects such as        tables, chairs, etc. at the region of interest;    -   the spatial covariance matrix R_(n) of each distance receiver is        calculated using the intra-frame multiple linear frequency        modulation, then R_(n) is inverted to obtain R_(n) ⁻¹, and the        upper diagonal of R_(n) ⁻¹ of each distance receiver is stored        in the memory, after which the output of the Capon beam former        is calculated for each distance receiver, and the angular        spectrum is stored in the memory to construct a [distance,        azimuth] heat map, and finally the [distance, azimuth] heat map        is transmitted to the Doppler estimation unit.

Object detection unit: a first channel in a distance domain and a secondchannel in an angle domain are processed using a Constant False-AlarmRate (CFAR) detection algorithm, and the second channel confirms theresult of the first channel, thereby removing clutter and noise, anddetermining a detection point.

Doppler estimation unit: for each [distance, azimuth] pair, the distancereceiver is filtered using the Capon beam weighting algorithm, and thenthe peak search is performed on the FFT of the filtered distancereceiver to estimate Doppler.

S102. Processing the first millimeter wave based on two types of timearrays and Doppler estimation to obtain at least one set of signalcharacteristic values corresponding to the first millimeter wave, whereeach set of signal characteristic values of the at least one set ofsignal characteristic values correspond to one frame of signal in thefirst millimeter wave.

After receiving the first millimeter wave through the millimeter waveapparatus, the terminal processes the first millimeter wave to obtain atleast one set of signal characteristic values corresponding to the firstmillimeter wave.

In the embodiment of the present application, after receiving the firstmillimeter wave, the terminal processes the first millimeter wave basedon the two types of time arrays to obtain a motion characteristiccorresponding to the gesture motion, where the motion characteristiccharacterizes a displacement information of the gesture motion;thereafter, the terminal extracts the at least one set of signalcharacteristic values from the motion characteristic based on theDoppler estimation, where each set of signal characteristic values ofthe at least one set of signal characteristic values correspond to oneframe of signal in the characterization of the motion characteristic.

In the embodiment of the present application, the two types of timearrays include a fast time array and a slow time array, and the terminalprocesses the first millimeter wave into at least one beam, where eachbeam of the at least one beam corresponds to a received first millimeterwave at one receiving time point; the terminal obtains at least onepiece of first information corresponding to the at least one beam in thefast time array, where the at least one piece of first informationcharacterizes at least one frequency corresponding to the at least onebeam; and thereafter, the terminal determines second informationaccording to the at least one piece of first information in the slowtime array, and the second information characterizes a frequency changebetween the at least one beam; and the second information is determinedas a motion characteristic.

In the embodiment of the present application, the terminal processes thefirst millimeter wave into at least one beam corresponding to eachreceiving time point, and the terminal calculates a frequencycorresponding to each beam of the at least one beam in the fast timearray, after which the terminal calculates the frequency change betweenthe at least one beam according to the frequency corresponding to eachbeam of the at least one beam in the slow time array, where thefrequency change characterizes the displacement information of thegesture motion, and the terminal determines the frequency change as themotion characteristic of the gesture motion.

It should be noted that the basic principle for the terminal torecognize different hand motions is: the hand is assumed to be adiscrete dynamic scattering center, and the Radio Frequency (RF)response of the hand is modeled as a superposition of responses from thediscrete dynamic scattering center; when the wavelength is smaller thanthe target spatial range, the scattering center model is consistent withthe geometrical theory of diffraction; due to the characteristic ofshort wavelength of the millimeter wave, the above assumption is appliedto the hand motion perception of the millimeter wave. This scheme adoptsa generalized time-varying scattering center model and considersnon-rigid hand dynamics, that is, each scattering center isparameterized via a composite reflectivity parameter and a radialdistance from a sensor, where the composite reflectivity parameter isfrequency dependent, which changes with the regional geometry of thehand with respect to the direction of the radar and the like. Therefore,the present application employs high temporal resolution perception,that is, the response of the hand to the radar is measured through ahigh frame rate, and then subtle temporal signal changes correspondingto these hand motions are extracted to detect subtle and complex handmotions. The terminal controls the millimeter wave apparatus to transmita periodic modulation waveform to implement the above concept, and themillimeter wave radar separately measures the corresponding receivedwaveform in each transmission cycle. Therefore, in order to implementthe above scheme, the present application defines two different timescales which are respectively short time scale perception and long timescale perception for analyzing the reflected first millimeter wave.

In the embodiment of the present application, the terminal uses shorttime scale perception in the fast time array and long time scaleperception in the slow time array.

In this case, the principle of short time scale perception is that thehigh radar repetition frequency links the scattering center hand modelwith the signal processing method, for a high velocity radar frequencywhich is high enough and a hand motion which is relatively slow, thescattering center model is approximately constant within a single radarrepetition interval, where the scattering center range and reflectivityare functions that closely follow the change of the short term scale T.The hand is illuminated with a single wide beam in each transmissioncycle, and all the scattering centers on the hand simultaneously reflectthe signal, and the measurement waveform consists of the reflection ofeach scattering center and is superimposed in the fast time, where eachindividual reflection waveform has instantaneous reflectivity and rangemodulation of the relevant scattering center; after RF demodulation andmodulation of specific filtered wave, the preprocessed received signalrepresents the superposition of the responses from respective scatteringcenters. The high radar repetition frequency is capable of, in the slowtime, capturing fine phase changes in the received signal correspondingdynamically to the scattering center.

In this case, the principle of long time scale perception is that whenthe scattering center moves, the relative displacement of the scatteringcenter may generate a phase change proportional to the wavelength. Thedependence of the phase change on the displacement allows the millimeterwave apparatus to find the scattered scattering center in the slow timeaccording to its phase. Assuming that the velocity of each scatteringcenter is approximately constant over some coherent processing timegreater than the radar repetition interval, the phase over the coherentprocessing time then generates a Doppler frequency, thus the Dopplerfrequencies of multiple scattering centers moving at differentvelocities can be analyzed by calculating the spectrum of the waveformof each fast time window over the coherent processing slow time window.

In the embodiment of the present application, the terminal processes thefirst millimeter wave into the motion characteristic corresponding tothe gesture motion using short time scale perception and long time scaleperception.

In the embodiment of the present application, a certain number ofconsecutive pre-processed radar signals are buffered in the fast timearray and the slow time array for characterizing the motioncharacteristic.

In the embodiment of the present application, each frame of signal iscomposed of at least 11 characteristic values. As shown in FIG. 3, the11 characteristic values include: num_detection, Doppler_average, andrange_average, magnitude_sum, positive num_detetion, range index,negative num_detection, negative doppler_average, range_display,angle_value and prediction_result.

In the embodiment of the present application, the Doppler effect is usedto calculate the velocity of the gesture motion and the Doppler shift,and the FMCW principle is used to calculate the distance from thegesture motion to the terminal.

S103. Identifying the at least one set of signal characteristic valuesusing a correspondence library of standard characteristic values andcontrol instructions, and obtaining a first control instructioncorresponding to the gesture motion.

After obtaining the at least one set of signal characteristic values,the terminal uses the correspondence library of standard characteristicvalues and control instructions to identify the at least one set ofsignal characteristic values, and obtains the first control instructioncorresponding to the gesture motion.

In the embodiment of the present application, the correspondence libraryof standard characteristic values and control instructions is arelational library obtained through the learning by a preset neuralnetwork, specifically, the terminal learns the standard gesture motionusing the preset neural network, and obtains at least one set ofstandard characteristic values corresponding to one control instruction;the terminal combines the control instruction and the corresponding atleast one set of standard characteristic values into the correspondencelibrary of standard characteristic values and control instructions, andafter the terminal obtains the at least one set of signal characteristicvalues corresponding to the first millimeter wave, the terminal searchesfor the first control instruction corresponding to the at least one setof signal characteristic values from the correspondence library ofstandard characteristic values and control instructions.

In an implementation, the preset neural network is a 6-layer residualnetwork obtained after removing the last three layers of a residualnetwork resnet18.

In the embodiment of the present application, after receiving thestandard gesture motion corresponding to each control instruction, theterminal processes the standard gesture motion to obtain a set of framesequence signals (standard frame signals) corresponding to the standardgesture motion, where each frame sequence signal within the set of framesequence signals corresponds to a set of characteristic values (a presetnumber of standard signal characteristic values), and the terminalinputs the at least one set of characteristic values corresponding tothe set of frame sequence signals into the 6-layer residual network, andlearns the at least one set of characteristic values by utilizing the6-layer residual network, and obtains a standard characteristic valuegroup corresponding to each control instruction, and saves the controlinstruction and the corresponding standard characteristic value group asa trained network model in .pkl format. When the terminal predicts a newgesture motion, the python script is invoked to import the trainednetwork model, and the script is invoked by a Matlab program, and thepython script returns a predicted classification result to the Matlabprogram after classifying and predicting the at least one set of signalcharacteristic values.

In the embodiment of the present application, the terminal matches theat least one set of signal characteristic values with the correspondencelibrary of standard characteristic values and control instructions, andwhen the at least one set of signal characteristic values successfullymatches with a first standard characteristic value group in thecorrespondence library of standard characteristic values and controlinstructions, the terminal searches for the first control instructioncorresponding to the first standard characteristic value group from thecorrespondence library of standard characteristic values and controlinstructions. At this time, the terminal obtains the first controlinstruction corresponding to the gesture motion using the correspondencelibrary of standard characteristic values and control instructions.

In the embodiment of the present application, the first controlinstruction is used to control the camera to implement the functions ofphotography, focusing, and the like, and the specific function isselected according to the actual situation, which is not specificallylimited in the embodiment of the present application.

Exemplarily, when the terminal receives the initial state of right handfingers naturally opened, the right forearm raised forward, and then theright arm elbow joint driving the forearm to lay flat toward the leftside with the elbow joint taken as the axis, and a change in the gestureof the right hand making a fist during the process of laying flat, theterminal determines the first control instruction as controlling thecamera to perform the photography.

S104. Controlling a first application to implement a correspondingfunction using the first control instruction.

After obtaining the first control instruction corresponding to thegesture motion, the terminal controls the first application to implementthe corresponding function using the first control instruction.

In the embodiment of the present application, after obtaining the firstcontrol instruction, the terminal inputs the first control instructioninto the Matlab program, and completes the function of controlling thecamera using the Matlab program. Specifically, the adopted manner thatthe terminal uses the Matlab program to complete the function ofcontrolling the camera is: the terminal invokes a Webcam module throughthe Matlab program; after the terminal obtains the first controlinstruction, the Matlab program transmits a control value correspondingto the first control instruction to the Webcam module; after receivingthe control value, the Webcam module controls the camera to implementdifferent functions according to different control values.

It should be noted that the Matlab program is used throughout the entiresystem. The Matlab program is used to store the signals collected by themillimeter wave apparatus, invoke the prediction script of Python forperforming the prediction, and control the camera after obtaining thepredicted value.

Exemplarily, as shown in FIG. 4, the overall architecture of the gesturecontrol is: the millimeter wave apparatus receives an original signalmodulated via the standard gesture motion, and processes the originalsignal to obtain at least one set of characteristic values, and then,inputs the at least one set of characteristic values into the neuralnetwork for analyzing, after the prediction of the neural network,controls the camera to complete the corresponding function.

It can be understood that the terminal receives, through the millimeterwave apparatus, the first millimeter wave returned by the gesturemotion, and processes the first millimeter wave based on the two typesof time arrays according to the characteristic of small wavelength ofthe first millimeter wave, and utilizes Doppler estimation to obtain theat least one set of signal characteristic values corresponding to theprocessed first millimeter wave, and finally obtains the first controlinstruction corresponding to the gesture motion using the at least oneset of signal characteristic values and the preset neural network,thereby the subtle gesture motion can be recognized, the accuracy ofgesture perception is thus improved.

Embodiment 2

The embodiment of the present application provides a method for gesturerecognition, which is applied to a terminal, and a millimeter waveapparatus is disposed on the terminal. As shown in FIG. 5, the methodmay include:

S201. Receiving, by a terminal, a reflected signal through themillimeter wave apparatus.

The method for gesture recognition provided by the embodiment of thepresent application is applied to a scenario in which a gesture of auser is perceived to implement a contactless photography.

In the embodiment of the present application, the terminal may transmita wireless signal through a transmitting antenna of the millimeter waveapparatus when a preset transmission time arrives, or the terminal maytransmit the wireless signal through the transmitting antenna of themillimeter wave apparatus when a first application launched, and thespecific timing at which the terminal transmits the wireless signalthrough the transmitting antenna of the millimeter wave apparatus isselected according to the actual situation, which is not specificallylimited in the embodiment of the present application.

In the embodiment of the present application, the first application is aphotographing application, a video shooting application, or the like,and is specifically selected according to the actual situation, which isnot specifically limited in the embodiment of the present application.

In the embodiment of the present application, when the user clicks theapplication icon of the first application in the display interface ofthe application icon, the terminal receives a startup instruction forstarting the first application, and at this time, the terminal startsthe first application, and uses the millimeter wave apparatus totransmit the wireless signal.

In the embodiment of the present application, the millimeter waveapparatus includes the transmitting antenna and a receiving antenna, andthe millimeter wave apparatus transmits the wireless signal using thetransmitting antenna, and forms a reflected signal after modulation viaa hand motion in a transmitting range of the wireless signal, and thenthe reflected signal is captured by the receiving antenna of themillimeter wave apparatus.

In the embodiment of the present application, the form of transmittingthe wireless signal by the millimeter wave apparatus is periodicallytransmitting Frequency Modulated Continuous Wave (FMCW), so that thefrequency changing rule of the first millimeter wave and that of thesecond millimeter wave are the same, which are both triangular waverule, however there is only a time difference in between, and theterminal can use this small time difference to calculate the targetdistance.

S202. Synthesizing, by the terminal, a reflected wave from the reflectedsignal using a beamforming algorithm.

After receiving the reflected signal through the millimeter waveapparatus, the terminal synthesizes the reflected wave from thereflected signal using the beamforming algorithm.

In the embodiment of the present application, after the receivingantenna receives the reflected wave, the reflected wave is cached intoan output cache area of an ADC, and then the millimeter wave apparatusmoves the reflected wave from the output cache area of the ADC to alocal memory of a DSP, and the terminal synthesizes the reflected wavefrom the reflected signal using a Capon beam former.

In the embodiment of the present application, a source signal isreconstructed from a sensor array using formula (1).X(t)=A(θ)s(t)+n(t)   (1)

-   -   where s(t) is an input signal after mixing the baseband signals.

In the embodiment of the present application, the static sundries isremoved by removing the DC components of each distance receiver in adistance processing module, thereby eliminating reflections of staticobjects such as tables or chairs, etc. at the region of interest.

S203. Removing, by the terminal, a clutter signal and a noise signal ofthe reflected wave to obtain a first millimeter wave, where the firstmillimeter wave is the reflected wave formed after a second millimeterwave transmitted by the millimeter wave apparatus is modulated via agesture motion.

After synthesizing the reflected wave from the reflected signal, theterminal removes the clutter signal and the noise signal in thereflected wave, thereby obtaining the first millimeter wave.

In the embodiment of the present application, a first channel in adistance domain and a second channel in an angle domain are processedusing a Constant False-Alarm Rate (CFAR) detection algorithm, and thesecond channel confirms the result of the first channel, therebyremoving clutter and noise, and determining the detection point, thusobtaining the first millimeter wave.

S204. Processing, by the terminal, the first millimeter wave into atleast one beam, where each beam of the at least one beam corresponds tothe first millimeter wave received at one receiving time point.

After obtaining the first millimeter wave, the terminal processes thefirst millimeter wave into at least one beam, where each beam of the atleast one beam corresponds to the first millimeter wave received at onereceiving time point.

In the embodiment of the present application, the terminal divides thefirst millimeter wave into at least one beam corresponding to at leastone receiving time point.

S205. Obtaining, by the terminal, at least one piece of firstinformation corresponding to the at least one beam in a fast time array,where the at least one piece of first information characterizes at leastone frequency corresponding to the at least one beam.

After processing the first millimeter wave into the at least one beam,the terminal obtains at least one piece of first informationcorresponding to the at least one beam in the fast time array.

In the embodiment of the present application, the terminal calculatesthe at least one piece of first information corresponding to the atleast one beam based on the principle of a short time scale perception,where the at least one piece of first information is used tocharacterize the at least one frequency corresponding to the at leastone beam.

In this case, the principle of short time scale perception is that thehigh radar repetition frequency links the scattering center hand modelwith the signal processing method, for a high velocity radar frequencywhich is high enough and a hand motion which is relatively slow, thescattering center model is approximately constant within a single radarrepetition interval, where the scattering center range and reflectivityare functions that closely follow the change of the short term scale T.The hand is illuminated with a single wide beam in each transmissioncycle, and all the scattering centers on the hand simultaneously reflectthe signal, and the measurement waveform consists of the reflection ofeach scattering center and is superimposed in the fast time, where eachindividual reflection waveform has instantaneous reflectivity and rangemodulation of the relevant scattering center; after RF demodulation andmodulation of specific filtered wave, the preprocessed received signalrepresents the superposition of the responses from respective scatteringcenters. The high radar repetition frequency is capable of, in the slowtime, capturing fine phase changes in the received signal correspondingdynamically to the scattering center.

S206. Determining, by the terminal, second information according to theat least one piece of first information in a slow time array, where thesecond information characterizes a frequency change between the at leastone beam.

After obtaining the at least one piece of first informationcorresponding to the at least one beam in the fast time array, theterminal determines the second information according to the at least onepiece of first information in the slow time array, where the secondinformation characterizes the frequency change between the at least onebeam.

In the embodiment of the present application, the terminal calculatesthe second information that characterizes the frequency change betweenthe at least one beam based on the principle of a long time scaleperception.

In this case, the principle of long time scale perception is that whenthe scattering center moves, the relative displacement of the scatteringcenter may generate a phase change proportional to the wavelength. Thedependence of the phase change on the displacement allows the millimeterwave apparatus to find the scattered scattering center in the slow timeaccording to its phase. Assuming that the velocity of each scatteringcenter is approximately constant over some coherent processing timegreater than the radar repetition interval, the phase over the coherentprocessing time then generates a Doppler frequency, thus the Dopplerfrequencies of multiple scattering centers moving at differentvelocities can be analyzed by calculating the spectrum of the waveformof each fast time window over the coherent processing slow time window.

Specifically, the terminal applies FFT to each fast time array over theslow time array to obtain a frequency information. The resulting fasttime-frequency mapping is converted to distance and velocity bytransformation. Fine adjustments can be made to the desired handdynamics and desired sensing performance based on SNR, velocityresolution, and Doppler aliasing. Therefore, a frequency changeinformation of the at least one beam is determined, where the frequencychange information characterizes the distance and rate of themulti-center of the hand which change over the time.

S207. Determining, by the terminal, the second information as the motioncharacteristic.

After determining the second information, the terminal determines thesecond information as a motion characteristic corresponding to the firstmillimeter wave.

In the embodiment of the present application, the terminal takes thedistance and rate of the multi-center of the hand which change over timeas a motion characteristic.

In the embodiment of the present application, the terminal buffers acertain number of consecutive pre-processed radar signals in the fasttime array and the slow time array for characterizing the motioncharacteristic.

S208. Sequentially extracting, by the terminal, velocity information andDoppler shift information corresponding to one frame of signal from themotion characteristic using the Doppler effect.

After determining the second information as the motion characteristic,the terminal extracts the velocity information and the Doppler shiftinformation corresponding to the one frame of signal from the motioncharacteristic using the Doppler effect.

In the embodiment of the present application, the terminal determinesthe motion characteristic as at least one frame of signal, and thensequentially processes one frame of signal out of the at least one frameof signal using the Doppler effect, so as to sequentially extract thecorresponding velocity information and Doppler shift information fromone frame of signal.

In the embodiment of the present application, the Doppler effect refersto that the wavelength of the object radiation changes due to therelative motion of the light source and the observer. In front of amoving wave source, the wave is compressed, the wavelength becomesshorter, and the frequency becomes higher; and in the back of the movingwave source, the opposite effect is generated, where the wavelengthbecomes longer, and the frequency becomes lower. The higher the velocityof the wave source, the greater the resulting effect is, a velocityinformation of the wave source motion in the direction of observationcan be calculated according to the degree of red/blue shift of the lightwave.

In the embodiment of the present application, since the millimeter waveapparatus transmits a frequency modulated continuous wave, the changerules of the frequency the second millimeter wave and the firstmillimeter wave are both in conformity with the triangular wave rule.Therefore, according to the Doppler effect, the frequency difference isshown in FIG. 6, where the solid line in the frequency-time coordinateis a frequency change curve of a transmitted wave, and the dashed lineis a frequency change curve of a received wave; fb is a frequencydifference when the detected object is stationary, and fd is the Dopplershift when the detected object moves.

S209. Sequentially extracting, by the terminal, distance informationcorresponding to one frame of signal from the motion characteristicusing a principle of frequency modulation continuous wave.

After determining the second information as the motion characteristic,the terminal sequentially extracts the distance informationcorresponding to the one frame of signal from the motion characteristicusing the principle of Frequency Modulation Continuous Wave.

In the embodiment of the present application, the terminal determinesthe motion characteristic as at least one frame of signal, and thensequentially processes one frame of signal of the at least one frame ofsignal using the principle of frequency modulation continuous wave tosequentially extract the corresponding distance information from oneframe of signal.

In the embodiment of the present application, the form in which themillimeter wave apparatus transmits the millimeter wave enables thecalculation of the distance between the relative targets. The basicprinciple is that the transmitted wave is a high-frequency continuouswave whose frequency changes with time according to the rule of thetriangle wave. The change rule of the frequency of the echo received bythe radar and the change rule of the frequency of the transmitted waveare the same, which are both triangular wave rule. However, there isonly a time difference in between, and the distance information can becalculated using this small time difference.

As shown in FIG. 7, the dashed line is the frequency change curve of atransmitted wave, and the solid line is the frequency change curve ofthe received wave, where td is the time difference between ft and fr, ftis the frequency of the transmitted wave, and fr is the frequency of thereceived wave.

S208 and S209 are two parallel steps after S207, and the specificexecution order is selected according to the actual situation, whichwill not be limited in the embodiment of the present application.

S210. Determining, by the terminal, at least the velocity information,the Doppler shift information, and the distance information as a set ofsignal characteristic values corresponding to one frame of signal.

After extracting the velocity information, the Doppler shiftinformation, and the distance information, the terminal determines atleast the velocity information, the Doppler shift information, and thedistance information as the set of signal characteristic valuescorresponding to one frame of signal.

In the embodiment of the present application, one frame of signalincludes a set of signal characteristic values related to the velocityinformation, the Doppler shift information, and the distanceinformation.

In the embodiment of the present application, each frame of signal iscomposed of at least 11 characteristic values related to the velocityinformation, the Doppler frequency shift information, and the distanceinformation. As shown in FIG. 3, the 11 characteristic values include:num_detection, Doppler_average, range_average, magnitude_sum, positivenum_detetion, range_index, negative num_detection, negativedoppler_average, range_disp, angle_value, and prediction_result.

S211. Combining, by the terminal, respective sets of signalcharacteristic values corresponding to respective frames of signals toform at least one set of signal characteristic values corresponding toat least one frame of signal.

After determining the set of signal characteristic values correspondingto the one frame of signal, the terminal combines respective sets ofsignal characteristic values corresponding to respective frames ofsignal to form the at least one set of signal characteristic valuescorresponding to the at least one frame of signal.

In the embodiment of the present application, the terminal sequentiallydetermines each set of signal characteristic values corresponding toeach frame of signal, and then, combines the respective sets of signalcharacteristic values corresponding to the respective frames of signalsto form the at least one set of signal characteristic valuescorresponding to at least one frame of signal.

S212. Identifying, by the terminal, the at least one set of signalcharacteristic values using a correspondence library of standardcharacteristic values and control instructions, and obtaining a firstcontrol instruction corresponding to the gesture motion.

After obtaining the correspondence library of standard characteristicvalues and control instructions, the terminal classifies and predictsthe at least one set of signal characteristic values using thecorrespondence library of standard characteristic values and controlinstructions.

In the embodiment of the present application, the terminal searches fora first standard signal characteristic value corresponding to the atleast one set of signal characteristic values from the correspondencelibrary of standard characteristic values and control instructions, anddetermines the first control instruction corresponding to the firststandard signal characteristic value, where the correspondence libraryof standard characteristic values and control instructions is arelational library obtained through the learning by a preset neuralnetwork.

In the embodiment of the present application, the terminal uses a Matlabprogram to invoke a python script, so as to import a trained networkmodel (the correspondence library of standard characteristic values andcontrol instructions). The python script returns the result value of theclassification and prediction to the Matlab program after classifyingand predicting the at least one set of signal characteristic values.

S213. Controlling, by the terminal, a first application to implement acorresponding function using the first control instruction.

After obtaining the first control instruction, the terminal controls thefirst application to implement the corresponding function using thefirst control instruction.

In the embodiment of the present application, after obtaining the firstcontrol instruction, the terminal inputs the first control instructioninto the Matlab program, and completes the function of controlling thecamera using the Matlab program. Specifically, the manner in which theterminal uses the Matlab program to complete the function of controllingthe camera is: the terminal invokes a Webcam module through the Matlabprogram; after the terminal obtains the first control instruction, theMatlab program transmits a control value corresponding to the firstcontrol instruction to the Webcam module; after receiving controlvalues, the Webcam module controls the camera to implement differentfunctions according to different control values.

It should be noted that the Matlab program is used throughout the entiresystem. The Matlab program is used to store the signals collected by themillimeter wave apparatus, invoke the prediction script of Python forperforming the prediction, and control the camera after obtaining thepredicted value.

Exemplarily, as shown in FIG. 9, the Matlab program is used throughoutthe entire system, and the millimeter wave apparatus transmits thecollected signal to the Matlab program; the Matlab program stores thesignal collected by the millimeter wave apparatus, invokes theprediction script of Python to perform prediction, and controls thecamera after obtaining the predicted value.

It can be understood that the terminal receives the first millimeterwave returned by the gesture motion through the millimeter waveapparatus, and processes the first millimeter wave based on the twotypes of time arrays according to the characteristic of small wavelengthof the first millimeter wave, and utilizes Doppler estimation to obtainthe at least one set of signal characteristic values corresponding tothe processed first millimeter wave, and finally obtains the firstcontrol instruction corresponding to the gesture motion using the atleast one set of signal characteristic values and the preset neuralnetwork, thereby subtle gesture motion can be recognized and theaccuracy of gesture perception is increased.

Based on the above embodiment 2, in the embodiment of the presentapplication, the above terminal further learns with the preset neuralnetwork in real time when performing gesture recognition, and obtainsthe correspondence library of standard characteristic values and controlinstructions, and the method for performing gesture recognition by theterminal may further include the following steps:

S301. Obtaining, by the terminal, a preset number of standard framesignals corresponding to a standard gesture motion.

In the embodiment of the present application, the terminal predeterminesa time period required for collecting one standard gesture motion, andthen determines the number of standard frame signals corresponding tothe time period.

Exemplarily, time period required for collecting one standard gesturemotion is 2 seconds, and the number of standard frame signals that canbe collected in 2 seconds is 60.

Exemplarily, the terminal obtains four standard gestures that controlthe camera to implement different functions, as shown in FIG. 10.

Gesture 1—Focusing: bending the middle finger, ring finger, and littlefinger of the right hand to make the fingertip touch the center of thepalm, extending the index finger and thumb to form a ellipse, separatingthe fingertip of the two fingers, opening the purlicue, forming a bevelangle with the forearm of the right hand and the ground, moving thewrist up and down to drive the right hand to make two knocks.

Gesture 2—Zooming in: making a fist, moving the hand forward andhorizontally toward the TI device and opening the fingers during themovement.

Gesture 3—Zooming out: opening the finger naturally, moving the handforward and toward the millimeter wave apparatus, and making a fistduring the movement.

Gesture 4—Photographing: naturally opening the fingers of the righthand, raising the forearm of the right hand forward, so that the backside of the forearm is directly facing the millimeter wave apparatus,and the right arm elbow joint driving the forearm to lay flat toward theleft side with the elbow joint taken as the axis, and making a fist bythe right hand during the process of laying flat.

S302. Determining, by the terminal, a preset number of standard signalcharacteristic values corresponding to the preset number of standardframe signals.

After obtaining a preset number of standard frame signals correspondingto the standard gesture motion, the terminal determines the presetnumber of standard signal characteristic values corresponding to thepreset number of standard frame signals.

In the embodiment of the present application, the terminal processes thestandard gesture motion to obtain a set of frame sequence signalscorresponding to the standard gesture motion, where each frame sequencesignal in the set of frame sequence signals corresponds to a set ofcharacteristic values.

Exemplarily, each frame sequence signal includes 11 characteristicvalues related to angle, distance, and Doppler shift.

S303: Performing, by the terminal, learning of the preset number ofstandard signal characteristic values using the preset neural network toobtain the correspondence library of standard characteristic values andcontrol instructions.

After determining the preset number of standard signal characteristicvalues corresponding to the preset number of standard frame signals, theterminal learns the preset number of standard signal characteristicvalues using the preset neural network, and obtains the correspondencelibrary of standard characteristic values and control instructions.

In an implementation, the preset neural network is a 6-layer residualnetwork obtained after removing the last three layers of the residualnetwork resnet18.

In the embodiment of the present application, the terminal inputs the atleast one set of characteristic values corresponding to the set of framesequence signals into the 6-layer residual network, and learns the atleast one set of characteristic values using the 6-layer residualnetwork, obtains the standard characteristic value group correspondingto each control instruction, and saves the control instruction and thecorresponding standard characteristic value group as the trained networkmodel in .pkl format.

Embodiment 3

FIG. 11 is a first schematic diagram of the composition and structure ofthe terminal according to an embodiment of the present application. Amillimeter wave apparatus is disposed on the terminal. In the practicalapplication, based on the same inventive concept of the first embodimentand the second embodiment, as shown in FIG. 11, the terminal 1 of theembodiment of the present application includes: a processor 10, areceiver 11, a memory 12, and a communication bus 13. In a process ofthe specific embodiment, the above processor 10 may be at least one ofan Application Specific Integrated Circuit (ASIC), a Digital SignalProcessor (DSP), or a Digital Signal Processing Device (DSPD), aProgrammable Logic Device (PLD), a Field Programmable Gate Array (FPGA),a CPU, a controller, a microcontroller, and a microprocessor. It is tobe understood that, for different devices, the electronic device forimplementing the functions of the above processor may be other device,which is not specifically limited in the embodiment of the presentapplication.

In the embodiment of the present application, the above communicationbus 13 is configured to implement the connection communication among theprocessor 10, the receiver 11 and the memory 12; the above receiver 11is configured to receive the first millimeter wave through themillimeter wave apparatus, where the first millimeter wave is thereflected wave formed after a second millimeter wave transmitted by themillimeter wave apparatus is modulated via a gesture motion; the aboveprocessor 10 is configured to execute an operating program stored in thememory 12 to implement the following steps:

-   -   processing the first millimeter wave based on two types of time        arrays and Doppler estimation and obtaining the at least one set        of signal characteristic values corresponding to the first        millimeter wave, where each set of signal characteristic values        of the at least one set of signal characteristic values        correspond to one frame of signal in the first millimeter wave;        identifying the at least one set of signal characteristic values        using the correspondence library of standard characteristic        values and control instructions, and obtaining the first control        instruction corresponding to the gesture motion; controlling the        first application to implement the corresponding function using        the first control instruction.

In the embodiment of the present application, the above processor 10 isfurther configured to process the first millimeter wave based on the twotypes of time arrays to obtain the motion characteristic correspondingto the gesture motion, where the motion characteristic characterizes thedisplacement information of the gesture motion; extract the at least oneset of signal characteristic values from the motion characteristic basedon the Doppler estimation, where each set of signal characteristicvalues of the at least one set of signal characteristic valuescorresponds to one frame of signal in the characterization of the motioncharacteristic.

In the embodiment of the present application, further, the two types oftime arrays include the fast time array and the slow time array; theabove processor 10 is further configured to process the first millimeterwave into at least one beam, where each beam of the at least one beamcorresponds to the first millimeter wave received at the receiving timepoint; obtain at least one piece of first information corresponding tothe at least one beam in the fast time array, where the at least onepiece of first information characterizes at least one frequencycorresponding to the at least one beam; determine the second informationaccording to the at least one piece of first information in the slowtime array, where the second information characterizes the frequencychange between the at least one beam; determine the second informationas the motion characteristic.

In the embodiment of the present application, further, the aboveprocessor 10 is further configured to sequentially extract the velocityinformation and the Doppler shift information corresponding to the oneframe of signal from the motion characteristic using the Doppler effect;sequentially extract the distance information corresponding to the oneframe of signal from the motion characteristic using the principle offrequency modulation continuous wave; and determine at least thevelocity information, the Doppler shift information, and the distanceinformation as the set of signal characteristic values corresponding tothe one frame of signal; combine respective sets of signalcharacteristic values corresponding to respective frames of signals toform the at least one set of signal characteristic values correspondingto the at least one frame of signal.

In the embodiment of the present application, further, the aboveprocessor 10 is further configured to obtain the preset number ofstandard frame signals corresponding to the standard gesture motion, anddetermine the preset number of standard signal characteristic valuescorresponding to the preset number of standard frame signals; learn thepreset number of standard signal characteristic values using the presetneural network to obtain the correspondence library of standardcharacteristic values and control instructions.

In the embodiment of the present application, further, the aboveprocessor 10 is further configured to receive the reflected signalthrough the millimeter wave apparatus; synthesize a reflected wave fromthe reflected signal into the using a beamforming algorithm; and removethe clutter signal and the noise signal of the reflected wave to obtainthe first millimeter wave.

In the embodiment of the present application, further, thecorrespondence library of standard characteristic values and controlinstructions is a relational library obtained through the learning bythe preset neural network.

The terminal proposed by the embodiments of the present applicationreceives the first millimeter wave through the millimeter waveapparatus, where the first millimeter wave is the reflected wave formedafter a second millimeter wave transmitted by the millimeter waveapparatus is modulated via a gesture motion; processes the firstmillimeter wave based on the two types of time arrays and the Dopplerestimation to obtain the at least one set of signal characteristicvalues corresponding to the first millimeter wave, where each set ofsignal characteristic values of the at least one set of signalcharacteristic values correspond to one frame of signal in the firstmillimeter wave; identifies the at least one set of signalcharacteristic values using the correspondence library of standardcharacteristic values and control instructions to obtain the firstcontrol instruction corresponding to the gesture motion; and controlsthe first application to implement the corresponding function using thefirst control instruction. It can be seen that, the terminal proposed bythe embodiment of the present application receives, through themillimeter wave apparatus, the first millimeter wave modulated via thegesture motion, and processes the first millimeter wave based on the twotypes of time arrays according to the characteristic of small wavelengthof the first millimeter wave, and obtains the at least one set of signalcharacteristic values corresponding to the processed first millimeterwave using the Doppler estimation, and finally obtains the first controlinstruction corresponding to the gesture motion using the at least oneset of signal characteristic values and the correspondence library ofstandard characteristic values and control instructions, thereby thesubtle gesture motion can be identified, and the accuracy of the gestureperception is improved.

The embodiment of the present application provides a storage medium,where the storage medium stores one or more programs, and the one ormore programs may be executed by one or more processors, and the storagemedium is applied in a terminal, and when the programs are executed bythe processor, the methods as described in the first embodiment and thesecond embodiment are implemented.

It is to be explained that the term “includes”, “including”, or anyother variants thereof, are intended to contain a non-exclusiveinclusion, such that a process, method, item, or system which include aseries of elements do not only include those elements, but also includeother elements that are not explicitly listed, or also include elementsthat are inherent to such process, method, item, or system. In the caseof no more limitations, an element defined by the phrase “including a .. . ” does not exclude the presence of additional identical elements ina process, method, item, or system that includes this element.

The serial numbers of the embodiments of the present application aremerely for the description, and do not represent the advantages anddisadvantages of the embodiments.

Through the description of the above implementations, those skilled inthe art can clearly understand that the foregoing embodiments of themethods can be implemented by means of software plus a necessary generalhardware platform, and of course, can also be implemented throughhardware, but in many cases, the former is the better implementation.Based on such understanding, the technical solution of the presentdisclosure, essentially or the part that contributes to the prior art,may be embodied in the form of a software product, where the softwareproduct of the computer is stored in a storage medium (such as theROM/RAM, disk, optical disk), and includes a number of instructions forcausing a terminal (which may be a cell phone, a computer, a server, anair conditioner, a network device, or the like) to perform the methodsdescribed in various embodiments of the present disclosure.

The embodiments of the present disclosure have been described above incombination with the drawings, but the present disclosure is not limitedto the specific embodiments described above, and the specificembodiments described above are merely illustrative and not restrictive,and under the enlightenment of the present disclosure. Those skilled inthe art can further make many forms without departing from the spirit ofthe present disclosure and the scope as claimed by the claims of thepresent disclosure, and these forms are all within the protection of thepresent disclosure.

INDUSTRIAL APPLICABILITY

In the embodiments of the present application, a terminal receives,through a millimeter wave apparatus, a first millimeter wave returned bya gesture motion, and processes the first millimeter wave based on twotypes of time arrays according to the characteristic of small wavelengthof the first millimeter wave, and obtains at least one set of signalcharacteristic values corresponding to the processed first millimeterwave using the Doppler estimation, and finally obtains a first controlinstruction corresponding to the gesture motion using the at least oneset of signal characteristic values and a preset neural network, therebysubtle gesture motion can be identified, and the accuracy of gestureperception is thus improved.

What is claimed is:
 1. A method for gesture recognition, wherein themethod is applied to a terminal, and the terminal is provided with amillimeter wave apparatus, the method comprising: receiving, through themillimeter wave apparatus, a first millimeter wave, wherein the firstmillimeter wave is a reflected wave formed after a second millimeterwave transmitted by the millimeter wave apparatus is modulated via agesture motion; processing the first millimeter wave based on two typesof time arrays and Doppler estimation to obtain at least one set ofsignal characteristic values corresponding to the first millimeter wave,wherein each set of signal characteristic values of the at least one setof signal characteristic values correspond to one frame of signal in thefirst millimeter wave; identifying the at least one set of signalcharacteristic values using a correspondence library of standardcharacteristic values and control instructions, and obtaining a firstcontrol instruction corresponding to the gesture motion; and controllinga first application to implement a corresponding function using thefirst control instruction.
 2. The method according to claim 1, whereinthe processing the first millimeter wave based on two types of timearrays and Doppler estimation to obtain at least one set of signalcharacteristic values corresponding to the first millimeter wavecomprises: processing the first millimeter wave based on the two typesof time arrays to obtain a motion characteristic corresponding to thegesture motion, wherein the motion characteristic characterizesdisplacement information of the gesture motion; extracting the at leastone set of signal characteristic values from the motion characteristicbased on the Doppler estimation, wherein each set of signalcharacteristic values of the at least one set of signal characteristicvalues correspond to one frame of signal of the characterization of themotion characteristic.
 3. The method according to claim 2, wherein thetwo types of time arrays comprise a fast time array and a slow timearray; the processing the first millimeter wave based on the two typesof time arrays to obtain a motion characteristic corresponding to thegesture motion comprises: processing the first millimeter wave into atleast one beam, wherein each beam of the at least one beam correspondsto the first millimeter wave received at one receiving time point;obtaining at least one piece of first information corresponding to theat least one beam in the fast time array, wherein the at least one pieceof first information characterizes at least one frequency correspondingto the at least one beam; determining second information according tothe at least one piece of first information in the slow time array,wherein the second information characterizes a frequency change betweenthe at least one beam; and determining the second information as themotion characteristic.
 4. The method according to claim 2, wherein theextracting the at least one set of signal characteristic values from themotion characteristic based on the Doppler estimation comprises:sequentially extracting velocity information and Doppler shiftinformation corresponding to the one frame of signal from the motioncharacteristic using a Doppler effect; sequentially extracting distanceinformation corresponding to the one frame of signal from the motioncharacteristic using a principle of frequency modulation continuouswave; determining at least the velocity information, the Doppler shiftinformation, and the distance information as one set of signalcharacteristic values corresponding to the one frame of signal; andcombining respective sets of signal characteristic values correspondingto respective frames of signals into the at least one set of signalcharacteristic values corresponding to the at least one frame of signal.5. The method according to claim 1, wherein the method furthercomprises: obtaining a preset number of standard frame signalscorresponding to a standard gesture motion; determining a preset numberof standard signal characteristic values corresponding to the presetnumber of standard frame signals; performing learning of the presetnumber of standard signal characteristic values using a preset neuralnetwork to obtain the correspondence library of standard characteristicvalues and control instructions.
 6. The method according to claim 1,wherein the receiving, through the millimeter wave apparatus, a firstmillimeter wave, comprises: receiving a reflected signal through themillimeter wave apparatus; synthesizing a reflected wave from thereflected signal using a beamforming algorithm; and removing a cluttersignal and a noise signal of the reflected wave to obtain the firstmillimeter wave.
 7. The method according to claim 1, wherein thecorrespondence library of standard characteristic values and controlinstructions is a relational library obtained through learning by apreset neural network.
 8. A terminal, comprising: a processor, areceiver, a memory, and a communication bus, wherein the terminal isprovided with a millimeter wave apparatus, and the receiver isconfigured to receive a first millimeter wave through the millimeterwave apparatus, wherein the first millimeter wave is a reflected waveformed after a second millimeter wave transmitted by the millimeter waveapparatus is modulated via a gesture motion; wherein the processor isconfigured to execute an operating program stored in the memory toimplement the following steps: processing the first millimeter wavebased on two types of time arrays and Doppler estimation to obtain atleast one set of signal characteristic values corresponding to the firstmillimeter wave, wherein each set of signal characteristic values of theat least one set of signal characteristic values correspond to one frameof signal in the first millimeter wave; identifying the at least one setof signal characteristic values using a correspondence library ofstandard characteristic values and control instructions, and obtaining afirst control instruction corresponding to the gesture motion; andcontrolling a first application to implement a corresponding functionusing the first control instruction.
 9. The terminal according to claim8, wherein the processor is further configured to: process the firstmillimeter wave based on the two types of time arrays to obtain a motioncharacteristic corresponding to the gesture motion, wherein the motioncharacteristic characterizes displacement information of the gesturemotion; and extract the at least one set of signal characteristic valuesfrom the motion characteristic based on the Doppler estimation, whereineach set of signal characteristic values of the at least one set ofsignal characteristic values correspond to one frame of signal of thecharacterization of the motion characteristic.
 10. The terminalaccording to claim 9, wherein the two types of time arrays comprise afast time array and a slow time array; the processor is furtherconfigured to: process the first millimeter wave into at least one beam,wherein each beam of the at least one beam corresponds to the firstmillimeter wave received at one receiving time point; obtain at leastone piece of first information corresponding to the at least one beam inthe fast time array, wherein the at least one piece of first informationcharacterizes at least one frequency corresponding to the at least onebeam; determine second information according to the at least one pieceof first information in the slow time array, wherein the secondinformation characterizes a frequency change between the at least onebeam; and determine the second information as the motion characteristic.11. The terminal according to claim 9, wherein the processor is furtherconfigured to: sequentially extract velocity information and Dopplershift information corresponding to the one frame of signal from themotion characteristic using a Doppler effect; sequentially extractdistance information corresponding to the one frame of signal from themotion characteristic using a principle of frequency modulationcontinuous wave; determine at least the velocity information, theDoppler shift information, and the distance information as one set ofsignal characteristic values corresponding to the one frame of signal;and combine respective sets of signal characteristic valuescorresponding to respective frames of signals into the at least one setof signal characteristic values corresponding to the at least one frameof signal.
 12. The terminal according to claim 8, wherein the processoris further configured to: obtain a preset number of standard framesignals corresponding to a standard gesture motion; determine a presetnumber of standard signal characteristic values corresponding to thepreset number of standard frame signals; and perform learning of thepreset number of standard signal characteristic values using a presetneural network to obtain the correspondence library of standardcharacteristic values and control instructions.
 13. The terminalaccording to claim 8, wherein the processor is further configured to:receive a reflected signal through the millimeter wave apparatus;synthesize a reflected wave from the reflected signal using abeamforming algorithm; and remove a clutter signal and a noise signal ofthe reflected wave to obtain the first millimeter wave.
 14. The terminalaccording to claim 8, wherein the correspondence library of standardcharacteristic values and control instructions is a relational libraryobtained through learning by a preset neural network.
 15. Anon-transitory storage medium, on which a computer program is stored,wherein the storage medium is applied to a terminal, and when executedby a processor, the computer program causes the terminal to perform thefollowing steps: receiving, through the millimeter wave apparatus, afirst millimeter wave, wherein the first millimeter wave is a reflectedwave formed after a second millimeter wave transmitted by the millimeterwave apparatus is modulated via a gesture motion; processing the firstmillimeter wave based on two types of time arrays and Doppler estimationto obtain at least one set of signal characteristic values correspondingto the first millimeter wave, wherein each set of signal characteristicvalues of the at least one set of signal characteristic valuescorrespond to one frame of signal in the first millimeter wave;identifying the at least one set of signal characteristic values using acorrespondence library of standard characteristic values and controlinstructions, and obtaining a first control instruction corresponding tothe gesture motion; and controlling a first application to implement acorresponding function using the first control instruction.
 16. Thenon-transitory storage medium according to claim 15, wherein theprocessing the first millimeter wave based on two types of time arraysand Doppler estimation to obtain at least one set of signalcharacteristic values corresponding to the first millimeter wavecomprises: processing the first millimeter wave based on the two typesof time arrays to obtain a motion characteristic corresponding to thegesture motion, wherein the motion characteristic characterizesdisplacement information of the gesture motion; extracting the at leastone set of signal characteristic values from the motion characteristicbased on the Doppler estimation, wherein each set of signalcharacteristic values of the at least one set of signal characteristicvalues correspond to one frame of signal of the characterization of themotion characteristic.
 17. The non-transitory storage medium accordingto claim 16, wherein the two types of time arrays comprise a fast timearray and a slow time array; the processing the first millimeter wavebased on the two types of time arrays to obtain a motion characteristiccorresponding to the gesture motion comprises: processing the firstmillimeter wave into at least one beam, wherein each beam of the atleast one beam corresponds to the first millimeter wave received at onereceiving time point; obtaining at least one piece of first informationcorresponding to the at least one beam in the fast time array, whereinthe at least one piece of first information characterizes at least onefrequency corresponding to the at least one beam; determining secondinformation according to the at least one piece of first information inthe slow time array, wherein the second information characterizes afrequency change between the at least one beam; and determining thesecond information as the motion characteristic.
 18. The non-transitorystorage medium according to claim 16, wherein the extracting the atleast one set of signal characteristic values from the motioncharacteristic based on the Doppler estimation comprises: sequentiallyextracting velocity information and Doppler shift informationcorresponding to the one frame of signal from the motion characteristicusing a Doppler effect; sequentially extracting distance informationcorresponding to the one frame of signal from the motion characteristicusing a principle of frequency modulation continuous wave; determiningat least the velocity information, the Doppler shift information, andthe distance information as one set of signal characteristic valuescorresponding to the one frame of signal; and combining respective setsof signal characteristic values corresponding to respective frames ofsignals into the at least one set of signal characteristic valuescorresponding to the at least one frame of signal.
 19. Thenon-transitory storage medium according to claim 15, wherein thecomputer program further causes the terminal to perform the followingsteps: obtaining a preset number of standard frame signals correspondingto a standard gesture motion; determining a preset number of standardsignal characteristic values corresponding to the preset number ofstandard frame signals; and performing learning of the preset number ofstandard signal characteristic values using a preset neural network toobtain the correspondence library of standard characteristic values andcontrol instructions.
 20. The non-transitory storage medium according toclaim 15, wherein the receiving, through the millimeter wave apparatus,a first millimeter wave, comprises: receiving a reflected signal throughthe millimeter wave apparatus; synthesizing a reflected wave from thereflected signal using a beamforming algorithm; and removing a cluttersignal and a noise signal of the reflected wave to obtain the firstmillimeter wave.